MIT researchers have found that, if a certain type of machine learning model is trained using an unbalanced dataset, the bias that it learns is …

If a machine-learning model is trained using an unbalanced dataset, such as one that contains far more images of people with lighter skin than people with darker skin, there is serious risk the model's predictions will be unfair when it is deployed in the real world. But this is only one part of the problem. MIT researchers have found that machine-learning models that are popular for image recognition tasks actually encode bias when trained on unbalanced data. This bias within the model is impossible to fix later on, even with state-of-the-art fairness-boosting techniques, and even when retraining the model with a balanced dataset. So, the researchers came up with a technique to introduce fairness directly into the model's internal representation itself.